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@ A proeeaa fdr eombtelng the resutta of several dassHtera, 



(g) A process for suocessftdy combining the re- 
sults of several classifiers provides a method for 
calculating confidenoes of each dassHicmton 
dedskm for every dassffier involved. Confi- 
dencas are then combined according to the 
Dempster-Shfifer Theory of Evidence. InWafly, 
basic probability assignments for each of the 
classifiers are calculated and used to calculate 
confidences for' each classifier. The confi- 
dences for afl of the dassHiers are then com- 
bined. The combined confidences are then 
used to detennine a class for the data input to 
the 
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Technical Ftetd 

Th present Invention Is Greeted to the field of pattern classification and recognition and, nwre partiou- 
iarly, to a mechanism for calculating a confidence of any classification decision. 

0 

Background Art 

Pattern recognKlon problein8» such as classification of machine or hand-printed character, are currently 
sohred with some accuracy ty using traditional classifiers or neural networks of different types. II is easier in 

fO many cases to apply several dassif iars to the same reoognitkm task to improve recognitk)n performance in a 
combined system^ Instead of inventing a new architecture or a feature extractor to achieve the same accuracy. 
However, It Is necessary to assign a measure of evidence to a classlficatkm dedston of each dassif ler in a 
system, to be able to combine them. Unfortunately, such assignments denwnd numerous approximatkms. ee* 
pedally when the number of dasses Is large. This creates computattorad difficulties, and can decrease the 

f 8 quality of the recognition performanoe. 

It is known bi the art to use the Dempster-Shafer Theory of EvMence as a tod for representing and cooh 
bining measures of evidence. In the existing art U.S. Patent No. 5,123,057 uses the Dempster-Shafer Theory 
of EvkJence to calculate a degree of match between data events portions and model parts. Evidence is col- 
lected and processed after preliminary matching has been performed using the Dempster-Shafer Theory of 

ap Evklenoe. 

Additk>nally. U.S. Patent Ho. 5,077,807 to Bokser describes a method for processing input feature vectora. 
Again, the '607 patent relates to a preprocessing means for pattern recognition. So although the prior art ad- 
dresses the problem of pattern dassif Icatfon and recognSfon, the existing art does not address the use of the 
Dempster-Shafer Theory of Evklence as a postreoognitton or postprocessing tool to combine results of seveiBl 

29 dassifiere. 

It Is seen then that It wouM be desirable to have a means for improving the results of dassif icatfon. 
Summary of the Inventton 

30 ' The present inventton improves the results of dassif icatk)n by successfully combining the results of sev* 
erel dassif iers. The Inventton also improves the results of dassif toatton by eliminating numerous approxima- 
Uons which result In a decreased accuracy and quality of the dassif icatlon and reoognitton performanoe. 

Spedf ically, the present Inventton uses s distance measure between a dassif ler oiitput vector and a mean 
vector for a subset of tralntog data corresponding to each dass. Using these distances for basto probability 
S8 assignments In the framework of the Dempster-Shaf^r Theory of EvUence. evtoences for all dassif icatton de- 
cistons for each dassifier can t>e calculated and combined. 

In accordance with one aspect of the present Inventton, a method for combining the results of several das- 
sif iere comprises a series of steps. Basto probability assignnrients are calculated fbr each of the dasaiftora 
and used to calculate oonf idenoes fdr each dassifier. The confidences for all of the dassif iere are then com- 
40 bined. Finally, the combined confUences are used to determine a dass for data Input to the dassif iers. 

Accordingly, it is an object of the present invention to provide a mechanism for calculating a conf toence 
of any dassif icatton deciston, which can increase the quality of recognitton performance. Other objects and 
advantages of the Inventton will be apparent from the fdlowing descriptton, the accompanying drawings and 
the appended daims. 

45 

Brtef Descriptton of the Drawings 

Fig. 1 1s block diagram illustreting the combination of several dassif Iers; and 
Fig. 2 is a flowchart illustrating the steps employed to achieve the combbiatton of the several dassif iere, 
60 as depteted in fig. 1. 

Detailed Descriptton of the Preferred Embodln>ents 

The present invention relates to a mechanism for combining the results of several dassif iers. Refening 
« to Fig. 1, a block diagram 10 illustrates the combination of several dassif iers. In Fig. 1. three dassif iere 12, 
1 4, and 16 are shown for descriptive purposes. However, the concept of the present inventton will be appitoaUe 
to e system having any number of dasslflere, from one to N. 

As can be seen in Fig. 1, data from data btod( 16 is input to each dassifier 12, 14, and 16. The present 
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invention allows for the determination of the ciaae of data from data block 18 j^etng input to the classifiers 12, 
14, and 16. based on the output of the dassifiera to a voter block 20. 

Refeiring now to Fig. 2 and cbntinuing with Fig. 1 . Inside each classifier block 12, 14, and 16. a basic prob- 
ability aaaignment is calculated, aa indtcated by block 22 of Fig. Z The baalc probabBity asslgnmenta are uaed 

5 to calculate confidences for each classifier, as indicated at block 24. The oonfUenoes output from each clas- 
sifier block are combined at the voter block 20. as Indicated at btock 26. The voter btock 20 uses the outputs 
of the classifiers as its input, can calculate confidence, i.e.. evidence, for the output of each classifier, and 
combines these confidences. The conf Idenoe or evidence for each classifier is a measure of the correctness 
of the answer that the classifier produces. As indicated at step 28 of Fig. 2. a classification is then made, Iden- 

fo tifying the Input from the data block 18 for the classifier which is to be recognized, based on the combined- 
confidences from the voter bk>ck 20. The output of the voter block 20, then, b the result of the classification 
process of the present Invention. The present invention provides improved results by taking into account the 
different decisions of different classifiers. 

In a preferred embodiment, the basic probability essignnwnts are calculated using a distance measure in 

15 accordance with the Dempster-Shafar Tlwory <rf Evidence. The confidences and the combinations are also 
calcdated aooording to the Dempstar-Shafer Theory of Evidence. Using the distance measures for basic prot>- 
ability assignments In the framework of the Dempster-Shafer Theory of Evidence, evidenoes fbr all dassif i- 
catton decisions for each dassif ier can be calculated and combined. 
In applying the teachings of the present inventton. assume 

» 

to be a sulMt of the training data corresponding to a dass k. In addition, assume 

to be a mean vector for a set 
^ for each dassif ier f» and each daaa k. Then 

is a reference vector for each daaa k. end 
^ is s distance measure bet%veen 

and 

This distance measure can be used to calculate the basic probability assignments of block 22 in Fig. 2. in ao- 
55 coniance with the Dempster-Shafer Theory of Evkience. 

Aooording to the Dempster-Shafar Theory of Evidence, consider a frame of discernment 

©»(6i..«.W. 

" vMre 

40 is the hypothesis that 'a vector 

? 

is of the dass k*. For any dassif ier f and each dass k. a distance measure d/> can represent evUenoe In sup> 
port of hypothesis 

45 If I sic and bi support Of 

or against 

if lis not equal to k. 

5^ With e as the frame of discernment 2^ denotes the set of all subsets of e. Afunctton m is called a basic 

probability assignment If m:2^ 

/n(e)"0 

10,1], and 
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wher rn{A) represents (he exsct belief in the hypothesis A. Therefore, if m is s basic probabiiay essignnient 
then a function Bei:2®-»(0,1] satisfying 

aeJ(B)-EiDM) 



is caOed a belief function. 

There is a one-to-one correspondence between the belief function and the basic probability assignment 
f 0 If mi and im are basic probability assignments on e. their oombtnation or orthogonal eum, 

Isdefinedas 

fs MA) =C-» E IB) nij (D) , 



O £ (H) m (D) , 



28 m{B) s 0 

and 

Since there is the one-to-one oorrespondeimrbetween Bel and m, the orthogonal sum of belief functions 

30 is defined in the obvious way. 

Special Idnds of belief functions are very good at representing evidence. TTieee functions ere called simple 
and separable support f unctiona. Bel is a simple support fimctfon if there exists en 

called the focua of Bet. stidi that Bel(e) » 1 and BeKA) » s, if both conditions, 
30 FgA 
and 

where s Is called Bel's degree of support 

Otherwise, Bel(A) » o. A separable support function Is either a simple support function or an orthogonal stm 
40 of simple support functions. Separable support functions are very useful when it is desired to combine evi- 
dences from several sources. If Bel Is a almple support function with fbous 

then m(F) « a, m(e) ■ 1«e, and m is 0 elsewhere. 

Let F be a focus for two simple support functions with degrses of support b> and S2. respecthwiy. If 
4S Bef 6e/f ® Ba^ 

then nr)(F) ■ 1 - (l-SiMI-Si). m(©) ■ (l-SiMI-sJ. snd m is zero elsewhere. 

Knowing these properties of the simple belief function, then df* can be used as a degree of support for 
the Bel with focus 

60 if 1 a k. Also, d^ are degrees of support for Bel with focus 

if I Is not equal to Ic This yields the probability assignnwnte 
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Combining all of the knowledge about focus, the evidence can be obtained fbr dass k and classifier n as: 
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Expanding on this equation yields: 



l-djt"(l-Il 

Finally. evWencee for an dassif iers may be combined according to the Dempster-Shafer rule to obtain a meas- 
ure of confidence for each dasa k for the input vector x» 

X. 

where 
Since 

la,l = i, 

In this case, then 



» ej,(x)=ne^<p) 



25 1^ 



Now, e dass m can be assigned to an input vector. 



in eocordance with the present Invention, there are two almost equivalent best functions fdr a distance 
measure between vector 

and 

>. 

One of these distance measures Is a oosfne between a classifier output vector and a mean vector for a 
subset of tralnbig data corresponding to each dass, or cos^Ok"), where Ok" Is an angle between 



40 



ot 



^ The other distance measure is a particular function of Eudtdean distance between a dassif ter output vector 
end a mean vector for a subset of training data corresponding to eech dass, or of Eudldian distance between 

and 
55 and: 
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5 

The present invention provides e siniple end effectrve method for cafodatlon of bask: protMbUity asslgiv 
ments. which permits confidences for indrvidual dassif ication dedskme to be obtained. The process and meth- 
od of the present invention can be used for the estimation of confidence for several classifiers and the conv 
10 binatlon of the results of several dassif iera. 

Industrial Applicability and Advantages 

The present invention is useful in the field of pattern classification and recognition and has the advantage 
15 of using the results of different classifiers to improve classification and recognition performance. The present 
invention has the further advantage of eliminating approximationa, which can be particularly prohibitive in cas- 
es where the number of classes is large. This. In turn, Increases the quality of the classification and recognition 
performance. Simple and effective calculation of basic probability assignments permits confidences for Indi- 
vidual classification decisions to be obtained. The present invention has the further advantage of allowtng for 
20 the estimation of confidenoe for several classifiers and the combination of the results of several classifiers. 

Finafly, the present inventton has the advantage of being applicable to any tnsKlitional statistical classifiers 
or neural networlcs of different architectures end based on different sets of featuras, aa weO es to different 
applications In which calculations of confidence for each classification decision are necessary. 

Having described the invention In detail and by reference to the preferred embodiment thereof, it will be 
25 apparent that other modifications and variations are possible without departing from the scope of the invention 
defined in the appended daims. 

The Invention is sunvnartzed as follows : 

1 . A method for combining resulte of several dassif iers comprises the steps of. 

calculating basic probability assignment ftor each of the dassif iers; 
30 using the basic probability assignments to calculate oonf Idenoes for eech classifier; 

combining the confidences fdr all of the dassif iers; and 

using the combined confidences to determine e dass for data Input to the dassif iers. 

2. A method eccording to 1 wherein the steps of cslculating bask: probability asalgnments end using the 
basic probability assignments to calculate confidences for each dassif ler comprise the step of applying 

35 a suitable theory of evidence. 

3. A method according to 2 wherein the suttaUe theory of evidence comprises the Dempster-Shafer Theory 
of Evidence. • - — • - 

4. A method according to 1 wherein the step of using the basic probability assignments to calculate con- 
fidences for each classifier comprises the steps of: 

40 using a distance measure between a dassif ler output vector and a mean vector for a subset of trairi- 

Ing data corresponding to each dass; and 

calculating evidences fdr ail classification decisions for each classifier, using the distances as basic 
probability assignmente. 

5. A method according to 4 wherein the distance meesure comprises one of two almost equivelent distance 
45 measures. 

6. A method according to 5 wherein the first distance measure comprises a cosine between a dassif ier 
output vector and a mean vector for a subset of training data corresponding to each dass. 

7. A method eooording to 5 wherein ttie second distance measure comprises a function of Euclidean die- 
tanoe between a dassif ier output vector and a mean vector fbr a subset of training data oonresponding to 

60 each dass. 

8. Amethod according to 1 wherein the step of combining the confidences comprises the step of combining 
the confidences according to the Dempster-Shafer Theory of Evidence. 

9. A system for combining the resulte of several dassif iers comprising: 

means for calculating basic probability assignments for each of the dassif iers; 
55 means for calculating confidences for each dassif ler from the basic probability assignments; 

means for combining the confidences for all of the dassif iers; and 

means for detennining a dass for data input to the dassif iers from the combined confidences. 

10. A system according to 9 wherein the means for calculating basic probability assignments and oonfl- 
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dencea for each dassif ier comprise a suitable theory of evidence. 

11. A system accoiding to 10 wherein the suitable theory of evidence comprises the Dempster-Shafer 
Theory of Evidence. 

12. A system according to 9 vi^herein the nwans for calculating confidences for each classifier comprises: 
5 a distance measure between a classifier output vector and a mean vector for a subset of training 

data corresponding to each dass; and 

means for calculating evidences for an dassif icatton decisions for each dassif iei; using the distanc- 
es as basic probability assignments. 

13. A system according to 12 vriierein the distance measure comprises one of two almost equivalent die- 
fo tance measures. 

14. Asystem according to 13 wherein the first distance measure comprises a cosine between a classifier 
output vector and a mean vector for a subset of training data corresponding to each dass. 

15. A system according to 13 wherein the second distance measure comprises e function of Eudidean dis- 
tance between a dassif ier output vector and a mean vector for a subset of training data corresponding to 

18 each dass. 



Claims 

» 1. A method for combining results of several dassif iers comprises the steps of: 
calculating basic probability assignments for each of the dassif iers; 
using the basic probability assignments to calculate confidences for each dassifier 
combining the conf idenoes fdr ail of the dassif iers; and 

uaing the combined conf idenoee to determine a dass for data input to the dassif iers. 

25 

2. A method as daimed in dahn 1 wherein the steps of calculating basic probaMtity assignments and using 
the basic probability assignments to catculato confidences for each classifier comprise the step of ap- 
plying a suitable theory of evidenoe. 

30 3. A method es daimed in daim 1 wherein the step of using the basic probability assignments to cdculato 
confidences for each dassifier comprises the steps of: 

using a distance measure between a dassif ier output vector and a mean vedorfbr a subset of train- 
ing date corresponding to each dass; and 

calculaUng evidences for all dassif icatton decisions for each dassif tor, using the distances as basic 
39 probability assignments. 

4. A method as daimed in daim 3 wherein the distance measure comprises one of two almost equivalent 
distance measures. 

5. A method as daimed in daim 4 wherein the f iret distance measure comprises a coaine between a daa- 
^ sif ier output vector and a mean vector for a subset of training date corre s ponding to each dass. 

6. A system for combining the results of several dassif lera comprising: 

means for calculating basic probabifity assignments for each of the dassif ierr, 
means for calculating confidences for each dassifier from the basic probability assignmenta; 
^ means for combining the confidences for all of the dassif iers; and 

means for determining a dass ftar date Input to the dassif lore from the combined confidences. 

7. A system as daimed in daim 6 wherein the means for calculating basic probability assignments and coiv 
f Idences for each classifier comprise a suitebto theory of evidenoe. 

so 

8. A system as daimed in daim 7 wherein the means fbr calculating confidences for each dassifier com- 
prises: 

a distance measure between a dassifier output vector and a mean vector for a subset of training 
date corresponding to each dass; and 
55 means for calculating evidenoea for aD dassif ication dedsiona for each dassifier. using the dis- 

tances as basic probability asstgnnienta. 

0. A system as daimed in daim 8 wherein the distence measure comprises one of two atmoat equivalent 
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distBnoB niB&sures. 

10. A system as claimed in dalm 9 wherein the second distance measure comprises a function of Eudidean 
distance between a classifier output vector and a mean vector for a subset of training data corresponding 
to each dass. 



fO 



fS 



30 



28 



SO 



39 



46 



48 



80 



8 



EP0621S56A2 




EP0621S56A2 



CALCULATE BASIC 
PROBAB/Urr 
ASSlGNyEHTS 



/ 



22 



CALCULATE 
CONFIDENCES 



/ 



24 



COMBINE 
CONFIDENCES 



/ 



26 



DETERMINE 
DATA CLASS 



/ 



2B 



FIG. 2 



10 



